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DESCRIPTION 

METHOD AND SYSTEM FOR CORRECTING KEYS TONING IN A PROJECTOR 
ARBITRARILY ORIENTED WITH RESPECT TO A DISPLAY SURFACE 

Technical Field 

This invention relates generally to image projectors, 
and more particularly, to correcting keystoning problems in 
projected images. 

Background Art 

Portable digital image projectors are common. These 
projectors, while connected to a PC or VCR, sit on a table and 
are aimed at a projection surface to show a "slide" presentation 
or a video. Many of these projectors use transmission LCDs, and 
typically only have a single main lens . The pro j ectors can display 
images one at the time or as a sequence of images. 

These projectors are typically designed so that level 
undistorted images are projected on the projection surface when 
the proj ector is placed horizontally on a level support surface, 
so that the projector's optical axis is lined up perpendicular 
to the, typically, vertical projection surface. If any of the 
above assumptions is violated, then the resulting image on the 
projection surface may not be rectangular and will be, at best, 
a trapezoid, and at worst an arbitrary quadrilateral. This 



1 



WO 03/017655 



PCT/JP02/07919 



problem is called keystoning. 

With prior art projectors, the only way to correct 
keystoning is to tediously adjust the physical position of the 
projector by moving it around, tilting and rotating it, until 
a near rectangular image is displayed. In some cases, it may 
be impossible to physically adjust the position of the pro j ector . 
For example, the projector may need to be well above or below 
the display surface. Some prior art projectors correct the 
distorted image optically or by the user providing projector 
positional data. 

U.S. Patent No. 5,548,357, "Keystoning and focus 
correction for an overhead projector," issued to Appel et. al. 
onAug. 18 1998, describes a system where a test slide is displayed. 
A user identifies line pairs most parallel to each other. The 
line pair identification activates a distortion correction 
program that uses the oblique angle betwe'en the horizontal plane 
through the projector and the viewing screen. 

U.S. PatentNo. 5,795, 046, "Method for pre-compensating 
an asymmetrical picture in a projection system for displaying 
a picture," issued to Woo on Aug. 1998, describes a system where 
the projection angle, and the trapezoidal error, is compensated 
for by the user entering in positional information into the system 
via a keyboard. 



2 



WO 03/017655 



PCT/JP02/07919 



Disclosure of Invention 

Amethod corrects keystoning in a projector arbitrarily 
oriented with respect to a display surface. An elevation angle, 
a roll angle, and an azimuth angle of an optical axis of the 
projector are measured with respect to the display surface. 

Aplanar projective transformation matrix is determined 
from the elevation, roll, and azimuth angles. A source image 
to be pro j ected by the pro j ector is warped according to the planar 
projective transformation, and then projected onto the display 
surface . 

Brief Description of Drawings 

Figure la is a illustration of a projector oriented 
perpendicularly to a planar display surface; 

Figure lb is a illustration of a projector oriented 
obliquely to the planar display surface; 

Figure 2 is a illustration of a projector according to 
the invention; 

Figures 3a-c illustrate three rotational angles 
associated with projectors; 

Figure 4 is a flow diagram of a warping function used 
by the invention. 4 is a schematic illustration of the image 
synthesis of the display system. 
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Best Mode for Carrying Out the Invention 

Figure la show a projector 100 placed on a table 10 and 
oriented at a display surface 101 so that a projected image 102 
is perfectly rectangular. This requires that the optical axis 
of the projector 100 be perfectly aligned perpendicularly to 
the plane of the display surface 101, and that there is no rotation 
about the optical axis. 

Figure lb shows a projector 103 placed obliquely on the 
table 10 and aimed at the display surface 101. In this case, 
a projected image 104 is some arbitrary shaped quadrilateral 
due to the oblique angle between the optical axis and the display 
surface . 

The present invention describes a method and system to 
correct the distorted image 104 to the required rectangular shape 
102 as seen in Figure 1 . As a result, this allows a user to casually 
place a projector on a surface, without being concerned if the 
table top is perfectly horizontal, and the projectors optical 
axis is perfect perpendicular to the display surface. 

As shown in Figures 3a-3b, when the projector is placed 
obliquely, or the table is titled, or the floor is tilted, or 
any combinations of these, the obliqueness can be expressed in 
terms of the three angles, elevation 301, roll 303, and azimuth 
305. Each angle is a measurement between the actual angle of 
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the optical axis 311 and the ideal angle 312 of the optical axis, 
i.e., perpendicular to the display surface and no rotation. In 
the ideal setup, as shown in Figure 1, all the three angles are 
zero . 

In the present invention, the values of these three angles 
301-303 are determined automatically using sensors mounted on 
the projector 200 as shown in Figure 2 . Figure 2 shows the projector 
200 according to the invention. The projector 200 includes 
multiple sensors, e.g., tilt sensor 201-202, and a camera 203. 
The sensors 201-202 can also be accelerometer implemented using 
2D-accelerometer boards, for example, the ADXL-2 02 from Analog 
Devices, Inc. The camera 203 canbe any digital camera with digital 
out-put signals. The camera 203 acquires an input image of a 
registration image having a predetermined pattern, e.g., a 
checkerboard pattern. These are well known. 

It shouldbe noted that the angles 301-303 shown in Figure 
3 can be sensed by other techniques such as lasers, magnetic 
sensors, or gyro sensors. With fewer sensors, e.g. , one or two, 
some keystoning effects canbe compensated. For example, a single 
sensor can correct for a non-zero elevation angle 301. 

The sensors 201-203 are coupled to a processor 210. The 
processor 210 is conventional in terms of its hardware and 
operating system software. Application software implements the 
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method 211 according to the invention. The method 4001 warps 
images, before they are projected, according to measurements 
taken by the sensors 201-203. The warping causes the otherwise 
arbitrary quadrilateral 104 to be projected appear as a 
rectangle . 

The elevation angle 301 and the roll angle 320 are 
independent of the geometric relationship between the projector 
200 and the display surface 101. Hence, these two angles can 
be sensed or measured using tilt or gravity sensors . For example, 
the sensor 201 is placed parallel to the optical axis of the 
projector, and the sensor 202 that measures the roll angle 302 
around the optical axis it is placed perpendicular to the optical 
axis of the projector. The azimuth angle 303 is determined by 
viewing a projected image with the camera 203. 

The azimuth angle 303 is derived from a single planar 
projective transformation (homography) between pixel locations 
in the projected image and corresponding pixel locations in an 
input image acquired by the camera 203 of the projected image, 
described in greater detail below. 

Figure 4 shows the steps of the warping 400 according 
to the invention. Typically, the computer application 211 
generates a rectangular image 400, having corners generally 
marked A, B, C, and D. The method 400 determines a suitable 
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quadrilateral 401, which when projected onto the display surface 
1091, appears as a correct displayed image 402, i.e., the image 
is rectangular and axis aligned with the real world sense of 
what is horizontal and vertical. 

The warping 400 scan converts the image 400 into the 
quadrilateral (warped) image 401 using the homography determined 
from the sensor measurements. The warping can be performed by 
texture mapping or image warping such as is available in 
conventional rendering engines. The homography is defined by 
the three angles 301-303. The warped image 401 is the input for 
the obliquelypositionedproj ector 200 according to the invention . 
The correct displayed image 402 is an axis aligned rectangle 
that fits inside the quadrilateral 104. In the shown example, 
the axes are horizontal and vertical directions. 

Rendering Process Given Three Angles using 3D Graphics 

The elevation and roll angles 301-302 are determined 
directly from the sensor 201-202. The azimuth angle 303 is 
determined by the following steps. Using the calibrated 
camera-proj ector pair, find the location of features in the input 
image acquired by the camera 203 of the registration image, e.g., 
corners or lines of the checkerboard pattern. The features are 
located in 3D in projector coordinate system. The projector 
coordinate systemhas a center of projection at the optical center 
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of the projector, and an optical axis along the positive z-axis 
of the projector coordinate system. 

The equation of a plane passing through at least three 
features is determined. The normal of this plane is N = [Nx, 
Ny, Nz] . A projection in the x-y plane is N xy = [Nx, Ny, 0] . 
The dot product of a normalized N xy and projector optical axis 
[0 0 1] is the cosine of the azimuth angle = cos" 1 (N xy . [0 0 
1] ) . After the three angles are determined, there are a number 
of ways of warping and rendering a correct image. Here, is a 
preferred method. 

The pro j ector illuminates an image with width iVandheight 
H at a distance D from the projector. This situation can be 
represented by a 3D coordinate system with the projector is at 
the origin and the illuminated rectangle on the display surface 
101 is parallel to the x-y plane at z = D. 

The extents of the projected quadrilateral are [W Lef t, 
W Rig ht] along x-dimension, and [H Bo ttom/ H Top ] along the y-dimension, 
W = I W Right - W Lef t I and H = | H Top - H Bo ttoml . This type of projection 
cone can be represented by a 3x3 perspective projection matrix 
P indicating the internal parameters of the proj ector that define 
the focal length and image center. 

The three angles elevation, roll, and azimuth correspond 
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to three 3x3 rotation matrices R e/ R r , R a . The goal is to render 
the source image texture mapped on a rectangle of size W x H. 
The rectangle is axis aligned in the x-y plane, and centered 
at the origin. The image is calculated from a rendering camera, 
with parameters P and the transformation defined below. This 
will give the required warping effect. A point [X, Y, 0] on the 
rectangle inx-y plane is mapped to apixel [x,y] with the following 
transformation and projection equation: 

[x y 1] ~= P* [ <R r * R e * R a *[X Y 0] T ) + [xs, ys, D] T , 
where xs = (W Left + W Ri g h t)/2, and ys = (H Bo ttom + H Top )/2. 

A second method achieves the same effect by warping the 
source image with a 3x3 homography matrix H between the 
corresponding set of points {m} and {n} . 

If 

[nx ny 1] T ~= P * [ <R r * R e * R a * ( [ (mx -xs) (my -ys) 
0] T ) + [xs, ys, D] T ), 
then the homography H is given by 

[nx ny 1] T ~= H * [mx my 0] T . 

Four or more pairs of {m = (mx, my) } and {n = (nx, ny) } 
are sufficient to determine the homography matrix H. A good set 
of choices for candidate {m} used in homography computation are 
the corners of the image rectangle WxH. 
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The invention effectively enables casual placement of 
a projector to generate correct imagery without distortion 
eliminating expensive electro-mechanical adjustments. The 
invention makes it possible to install a projector in any 
orientation e.g. upside down or resting on its side, and detect 
the parameters of correct imagery by adding sensors along various 
directions . 

This invention is described using specific terms and 
examples. It is to be understood that various other adaptations 
and modifications may be made within the spirit and scope of 
the invention. Therefore, it is the object of the appended claims 
to cover all such variations and modifications as come within 
the true spirit and scope of the invention. 
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CLAIMS 

1. A method for correcting keystoning in a projector 
arbitrarily oriented with respect to a display surface, 
comprising; 

measuring an elevation angle, a roll angle, and an azimuth 
angle of an optical axis of the projector with respect to the 
display surface; 

determining a planar projective transformation matrix 
from the elevation, roll, and azimuth angles; 

warping a source image to be projected by the projector 
according to the planar projective transformation; and 

projecting the warped source image onto the display 

surface . 

2 . The method of claim 1 wherein the elevation and roll 
angels are measured by tilt sensors, and the azimuth angle is 
a camera coupled to the projector. 

3. The method of claim 2, further comprising: 
acquiring an input image of a registration image; and 
determining the azimuth angle from the input image. 

4. The method of claim 1 further comprising: 
measuring the elevation angle with a tilt sensor. 
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5. The method of claim 1 further comprising: 
measuring the roll angle with a tilt sensor. 



6. The method of claim 1 further comprising: 
measuring the azimuth angle with a camera-projector 

pair. 



7. The method of claim 1 wherein the projector uses a 
laser beam. 



8. A system for correcting keystoning in a projector 
arbitrarily oriented with respect to a display surface, 
comprising; 

a first tilt sensor measuring an elevation angle of an 
optical axis of the projector with respect to the display surf ace; 

a second tilt sensor measuring an roll angle of an optical 
axis of the projector with respect to the display surface; 

a camera coupled to the projector measuring an azimuth 
angle of an optical axis of the projector with respect to the 
display surface; 

means for determining a planar projective 
transformation matrix from the elevation, roll, and azimuth 
angles; and 

means for warping a source image to be projected by the 
pro j ector according to the planar proj ective transformation and 
projecting the warped source image onto the display surface with 
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